On the Role of Statistical Significance in Exploratory Data Analysis

Itece~tl~, an approach to k~owldge d~co~er~, called A~ribute Focusing, ham been used by.soflwa~ development teams to discover ~¢h ~owledge categorical defeef data as allows them to imFeo~e ~eir process of software development in real Sime. This feedback is provided by computing the difference o] the obme~ed propo~ion wiShin a selected ea~ego,j ~om an effipected proportion for that category, and then, by siudlling those di.ffe_rences to iden. Sift,/their causes in She co¢~ezt of She process and product. In this paper, we consider the possibility that some differencem may #im~ly ha~e oceu~red b~/ chance, i.e., as a co~ce of some random effect in the proceu 9races Sing the data. We de~elo? an approach based on statistioal significance to identify such differences. Preliminary, empirical results are p~esented ~hich iedicate that kno~oledge of statistical sip~ificance should be used carnally ~hen selecti¢$ differences to be studied So identify causes. Conventional ~dOm would suggest Shat all differences that lie bsyvnd some small l~el of statistical significance be eliminated from con~i~eraSion. O~r r~sul~ show that such elimination is not a 9ood idea. Tl~ abo ~how that information o~ statistical significance can be useful in the process of identifyin9 a cause.